Skip to content

Cosmos: downgrade request-status error logs to debug - #4711

Open
analogrelay with Copilot wants to merge 5 commits into
mainfrom
copilot/fix-cosmos-tracing-levels
Open

Cosmos: downgrade request-status error logs to debug#4711
analogrelay with Copilot wants to merge 5 commits into
mainfrom
copilot/fix-cosmos-tracing-levels

Conversation

Copilot AI commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

Create-if-not-exists and similar flows can intentionally treat non-2xx service responses (for example, 409 Conflict) as successful control flow. The Cosmos pipeline was emitting high-severity logs for these outcomes, creating noisy WARN/ERROR output under normal usage.

  • Pipeline abort logging severity

    • Changed operation_pipeline abort-path logging from tracing::error! to tracing::debug! for terminal operation outcomes that include typed Cosmos HTTP status.
  • Hedged terminal HTTP outcome severity

    • Changed hedged terminal HTTP status logging (cosmos.hedge.terminal_http_error) from tracing::warn! to tracing::debug!.
  • Behavioral impact

    • Status/sub-status telemetry remains logged, but request-status messages no longer surface as warn/error by default for user-handled HTTP outcomes.
// before
tracing::error!(status = ?cosmos_status, "operation aborted");
tracing::warn!(http_status = u16::from(status.status_code()), "cosmos.hedge.terminal_http_error");

// after
tracing::debug!(status = ?cosmos_status, "operation aborted");
tracing::debug!(http_status = u16::from(status.status_code()), "cosmos.hedge.terminal_http_error");

Co-authored-by: analogrelay <7574+analogrelay@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix logging of user-triggered errors in Cosmos DB Cosmos: downgrade request-status error logs to debug Jul 7, 2026
Copilot AI requested a review from analogrelay July 7, 2026 22:31
@analogrelay
analogrelay marked this pull request as ready for review July 24, 2026 18:48
@analogrelay
analogrelay requested a review from a team as a code owner July 24, 2026 18:48
Copilot AI review requested due to automatic review settings July 24, 2026 18:48
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Downgrades Cosmos HTTP-status logging to reduce noise from expected control-flow responses.

Changes:

  • Changes pipeline abort logs from error to debug.
  • Changes hedged terminal HTTP logs from warning to debug.
Show a summary per file
File Description
operation_pipeline.rs Lowers terminal outcome log severity.

Review details

  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Medium

@analogrelay analogrelay left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot While you're fixing up the issues below, look out for any places where we're tracing "event name"-style strings like cosmos.hedge.terminal_http_error. Tracing events should have human-readable log-line-style messages.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot This one should probably be debug! as well.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 16ae25a. This log is now emitted at debug!.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot Fix this up to use a human-readable log line instead of an "event name" style. Something like non-retriable http error in hedging attempt

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 16ae25a. This now uses a human-readable message: non-retriable http error in hedging attempt.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot Fix this up to use a human-readable log line instead of an "event name" style. Something like non-retriable transport error in hedging attempt

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated in 16ae25a. This now uses a human-readable message: non-retriable transport error in hedging attempt.

Co-authored-by: analogrelay <7574+analogrelay@users.noreply.github.com>
Copilot AI requested review from Copilot and removed request for Copilot July 24, 2026 19:07

Copilot AI commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

@copilot While you're fixing up the issues below, look out for any places where we're tracing "event name"-style strings like cosmos.hedge.terminal_http_error. Tracing events should have human-reada...

Addressed in 16ae25a. Updated the touched hedging terminal tracing events to use human-readable log-line messages instead of event-name-style strings.

Copilot AI requested a review from analogrelay July 24, 2026 19:08
@analogrelay
analogrelay dismissed their stale review July 27, 2026 20:46

No longer relevant

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need a changelog entry here?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additionally, I think this might introduce some conflict with the specs.

§10.4's level table and OTel-naming note are now partially contradicted.

request_count = diagnostics.request_count(),
http_status = u16::from(status.status_code()),
sub_status = ?status.sub_status(),
"cosmos.hedge.terminal_http_error",

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No migration note for anyone alerting on cosmos.hedge.terminal_http_error — string is gone. Breaking for log-based alerting, or never a supported surface? (§10.4 says "reserved," which cuts toward the latter.)

@analogrelay

Copy link
Copy Markdown
Member

@copilot Can you address @NaluTripician 's two comments? Specifically:

Do we need a changelog entry here?

No migration note for anyone alerting on cosmos.hedge.terminal_http_error — string is gone. Breaking for log-based alerting, or never a supported surface? (§10.4 says "reserved," which cuts toward the latter.)

Yes, but we don't need detail. It can simply say "Changes the messages and severity of some tracing events". It's early enough that I'm not worried about being hyper-accurate there. Add that CHANGELOG entry (ensure you reference this PR as all the others do)

Additionally, I think this might introduce some conflict with the specs.

Update relevant hedging specs for the change in event text.

Co-authored-by: analogrelay <7574+analogrelay@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 28, 2026 21:04
Copilot AI removed the request for review from Copilot July 28, 2026 21:04

Copilot AI commented Jul 28, 2026

Copy link
Copy Markdown
Contributor Author

@copilot Can you address @NaluTripician 's two comments? Specifically:

Do we need a changelog entry here?
...

Addressed in 74c6479.

  • Added a Cosmos driver changelog entry under 0.38.0 (Unreleased) with the requested wording and PR reference.
  • Updated HEDGING_SPEC.md §10.4 to align with the current human-readable hedging terminal log text and clarify that those terminal messages/levels are outside the reserved telemetry-name contract.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cosmos: Avoid tracing "WARN" or "ERROR" messages for user-triggered errors

4 participants